Information geometry of the EM and em algorithms for neural networks

In order to realize an input-output relation given by noise-contaminated examples, it is e ective to use a stochastic model of neural networks. A model network includes hidden units whose activation values are not speci ed nor observed. It is useful to estimate the hidden variables from the observed or speci ed input-output data based on the stochastic model. Two algorithms, the EM and em-algorithms, have so far been proposed for this purpose. The EM -algorithm is an iterative statistical technique of using the conditional expectation, and the em-algorithm is a geometrical one given by information geometry. The em-algorithm minimizes iteratively the Kullback-Leibler divergence in the manifold of neural networks. These two algorithms are equivalent in most cases. The present paper gives a uni ed information geometrical framework for studying stochastic models of neural networks, by forcussing on the EM and em algorithms, and proves a condition which guarantees their equivalence. Examples include 1) Boltzmann machines with hidden units, 2) mixtures of experts, 3) stochastic multilayer perceptron, 4) normal mixture model, 5) hidden Markov model, among others. key words: EM algorithm, information geometry, stochastic model of neural networks, learning, identi cation of neural network, e-projection, m-projection, hidden variable